Programming contact tasks using a reality-based virtual environment integrated with vision

We present an integrated system in which an operator uses a simulated environment to program part-mating and contact tasks. Generation of models within this virtual environment is facilitated using a fast, occlusion-tolerant, 3D grey-scale vision system which can recognize and accurately locate objects within the work site. A major goal of this work is to make robotic programming easy and intuitive for untrained users working with standard desktop hardware. Simulation can help accomplish this, offering the ease-of-use benefits of “programming by demonstration”, coupled with the ability to create a programmer-friendly virtual environment. Within a simulated environment, it is also straightforward to track and interpret an operator’s actions. The simulator models objects as polyhedra and implements full 3D contact dynamics, making is easy to place and manipulate objects using input from a simple 2D mouse. When a manipulation task is completed, local planning techniques are used to turn the virtual environment’s motion sequence history into a set of robot motion commands capable of realizing the prescribed task.